20.11 Drawing Samples from Autoencoders

• Some kinds of autoencoders, such as variational autoencoders, explicitly represent a probability distribution and admit straightforward ancestral sampling.
• Most other kinds of autoencoders require MCMC sampling.

20.11.1 Marlove Chain Associated with Any Denoising Autoencoder

Generalized denoising autoencoders aare specified by a denoising distribution for sampling an estimate of the clean input given the corrupted input. Each step of the Markov chain that generates from the estimated distribution consists of the following step:

1. Starting from the previous state x, inject corruption noise, sampling \(\hat{x}\) from \(C(\hat{x} | x)\)
2. Encode \(\hat{x}\) into \(h=f(\hat{x})\)
3. Decode h to obtain the parameters \(w=g(h)\) of \(p(x|w = g(h)) = p(x|\hat{x})\)
4. Sample the next state x from \(p(x|w = g(h)) = p(x|\hat{x})\)

20.11.2 Clamping the Conditional Sampling

Similar to Boltzmann Machine, denoising autoencoders and their generalizations can be used to sample from a conditional ditribution \(p(x_f|x_o)\), simply by clamping the observed units \(x_o\) and resampling the free units \(x_f\) given \(x_o\) and the sampled latent variables (if any).

20.11.3 Walk-Back Training Procedure

The walk-back training procedure was proposed as a way to accelerate the convergence of generative training of denoising autoencoders. It consists of multiple stochastic encoder-decoder step, initialized at a training exmple and penalizing the last probablistic reconstruction (or all the reconstructions along the way. )